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ABSTRACT 

The report explains the major design concepts of the 
Reporting and Evaluation System for Occupational Education (RESOE) 
and provides a framework for the system's utilization by New York 
State's occupational directors and guidance personnel. Following an 
introduction the second section summarizes the functional 
specifications and general design concepts for RESOE which were 
determined during the first year of RESOE "s development. An outline 
of the general strategy for implementing the system at regional ana 
local education agencies throughout i.he State is followed by a 
detailed description of the way in which the general implementation 
strategy was carried out in two of New York State's occupational 
education planning regions. Section 5 describes a feasibility study 
and field test for the iinplementation of RESOE in New York City. The 
final section of the report discusses some of the requirements for 
the sustained implementation of RESOE in planning regions where 
initial implementation has already taken place^ as well as in 
planning regions where RESOE has not yet been implemented. Emphasis 
is placed on local uses of the system for reporting and evaluation 
proce sses • (A ut hor/JR) 
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Preface 

During the past year, a Reporting and Evaluation System 
for Occupational Education (RESOE) has been further developed 
and implemented in two of New York State's occupational 
planning regions. Flexible computer-maintained student files 
have been established for each participating education agency. 
These files v/ere subsequently utilized to generate mandated 
reports for the state and federal governments, and are avail- 
able to meet other local, state and federal reporting and 
evaluation requirements in occupational education. The 
feasibility of applying RESOE in New York City's academic 
and vocational high schools was successfully demonstrated in 
a system field test. In addition, a layman's guide to RESOE 
was developed. This guide explains RESOE ' s major design 
concepts and provides a framework for system utilization by 
the state's occupational directors and guidance personnel. 

More important than those tangible results is the manner 
in which RESOE has been received by occupational directors and 
their guidance personnel. They have not looked upon RESOE as 
simply another means of fulfilling State Education Departmen*: 
reporting requirements, but have generally appreciated the 
underlying system concepts described in this report. Some 
occupational educators* have already sought to "get more mileage" 
out of the system to meet their own local or regional reporting. 
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evaluation and management information requirements. They have 
appreciated that the system is here now, and ready to respond 
to many of their needs . 

Many occupational educators throughout New York State 
have graciously supported the efforts of the State Education 
Department's Office of Occupational Education and Riverside 
Research Institute (RRI) to develop and implement RESOE. 
Their commitment to RESOE 's implementation in their respective 
education agencies resulted in an orderly implementation process. 
Their willingness to specify their management information re- 
quirements, and to provide constructive criticism, has enabled 
RRI to further elaborate RESOE so that the needs of education 
agencies can be served better • 

It is not possible to mention all of the occupational 
administrators, teachers, and other personnel who have supported 
the implementation of RESOE. However, RRI wishes to acknowledge 
the substantial contributions of the occupational administrators 
and guidance coordinators who were designated in their education 
agencies to work directly with RRI personnel during the system 
implementation period. 
• Syracuse 

Mr. Hans Lang, Director of Occupational and Continuing 
Education 

Mr. Donald Caldera, Coordinator for Occucational 
Education Curriculum Development 

Mr. Vincent Brennan, Assistant Special Needs and Work 
Study Program 
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• Tompkins-Seneca-Tioga BOCES 

Mr. Duane Ash, Guidance Coordinator 

• Cayuga BOCES 

Mr. Clifford Cole, Director for Adult Education 

• Oswego BOCES 

Mr. Ronald Service, Guidance Coordinator 
Mr. Paul Bradshaw, Guidance Coordinator 

• Cortland-Madison BOCES 

Mr. Francis Streeter, Director of Occupational Education 
Mr. Raymond Cook, Director of Adult Education 

• Onondaga BOCES 

Mr. Patrick Shaughnessy, Assistant Superintendent 
Mrs. Lois Wilson, Assistant 



• Nassau County BOCES 

Mr. Frank Russo, Supervisor of Occupational Education 
Mr. James Wilde, Program Associate of Occupational 

Education 

• Nassau County Vocational Education Extension Board 

Mr. Don Hoake, Director 

Mr. Al Fieri, Coordinator of Veterans' Training 

• New York City Board of Education 

Mr. George Quarles, Chief Administrator, Office of 
Career Education 

Mr. Norman Elliott, Director (Acting) Bureau of Business 
and Distributive Education 
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I. Introduction 

The Vocational Education Amendments of 196 8 3et forth 
new national objectives for occupational education. Not 
only was occupational education to serve the industries 
which needed manpower; but, more significantly, it was to 
serve all students who needed to acquire vocational skills 
and knowledge • Subsequent to VEA 68/ enrollments in 
occupational programs have steadily grov-m, and occupational 
studies now occupy a pivotal position in American education* 

At the secondary and college levels, there is increasing 
recognition that students should supplement traditional 
academic subjects with the acquisition of vocational skills 
and knowledge that will sustain them in the world of work* 
Publicly supported occupational education has also rapidly 
expanded its services to adult members of the labor force, 
providing training, and retraining for new employment and 
advancement* Special student groups, such as the urban 
disadvantaged, the rural disadvantaged, and children of 
retarded mental development, are receiving career-oriented 
education, so that these groups may be more readily assimi- 
lated into the labor force. Finally, occupational education, 
to a greater extent than any other form of education, is 
expected to help in minimizing unemployment, and to provide 
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a sufficient labor supply for various sectors of New York 
State's economy. 

In a recent policy statement, the New York State Regents 
have called for schools to instill greater career conscious- 
ness throughout the entire educational system and to offer 
more comprehensive occupational programs and services. 
Occupational education will be expected to play a key role in 
New York State to help to guarantee productive employment 
for all who are able and willing to work. 

In order to develop a more comprehensive and coordinated 
system of occupational education which serves the needs of 
all students as well as the labor requirements of Nev; York 
State's economy, an occupational education information syr.tem 
is needed. The Regents have said that " . , , state, regional, 
and local administrators will have access to standardized 
management information systems which provide all data needed 
to target, manage, and evaluate occupational education 
efforts; e.g., data concerning target groups, enrollments, 
program effectiveness, costs, and manpower needs. Such 
management information systems will ensure accouncability 
and constant feedback for program redesign." 



Regents of the State University of the State of New York, 
Occupational Education ; a Statement of Policy and Proposed 
Action . Albany : the State Education Department, May, 1971. 
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For the past two years, Riverside Research Institute 

(RRI) has been working with the State Education Department 

(SED) to develop and implement a system which will facilitate 

the development and maintenance of more comprehensive 

occupational services for students. This system is called 

The Reporting and Evaluation System for Occupational Education 

(RESOE) . This volume of RRI*s final report for 1972 describes 

RESOE, and the progress that has been made in its implementa- 
* 

tion . 

This volume is organized into six sections. The next 
section (Section II) summarizes the functional specifications 
and general design concepts for RESOE which were jointly 
determined by SED and RRI during the first year of RESOE 's 
development. Section III describes the general strategy for 
implementing the system at regional and local education 
agencies throughout the state. Section IV describes how the 
general implementation strategy was carried out in two of 
New York State's occupational education planning regions. 
This section is relatively detailed, and may be bypassed by 
the reader who is more interested in system concepts than 
in the details of system operation. Section V describes a 

This volume is submitted under contract #C-53226 between 
Riverside Research Institute and the New York State Education 
Department. This written report supplements a final oral 
report delivered by Riverside personnel at the State Education 
Department on December 19, 19 72. 
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feasibility study and field test for the implementation of 
RESOE in New York City. This section consists of three 
parts: a detailed review and critique of New York City's 
current procedures for accomplishing some of the objectives 
met by RESOE; a plan for the implementation of RESOF in New 
York City; and a field test of certain parts of this imple- 
mentation plan. 

The final section of the report discusses some of the 
requirements for the sustained implementation of RESOE in 
planning regions where initial implementation has already 
taken place, as well as in planning regions where RESOE has 
not yet been implemented. The section stresses the local 
use of the system, and emphasizes the utilization of RESOE 
for local and regional reporting -^nd evaluation processes 
which are not required by the State Education Department. 



F/242-5-00 



- 4 - 



II . Background 

During the initial year of system development (1971) , 
RRI conferred with the State Education Department's Division 
of Occupational Education Planning and carried out a systems 
study of occupational education at the Nassau County Board 
of Cooperative Education Services (BOCES) . This work led to 
a preliminary set of functional specifications for RESOE. 
Basic system design concepts were developed during the first 
year and partially tested at the Nassau BOCES. This section 
describes the functional specifications that were formulated 
for RESOE and describes the basic system design concepts 
that meet the specifications. 
A. Functional Specifications 

1. The system must be capable of supporting the 
evaluation of occupational education programs^ and of 
providing feedback for program design and redesign . 

a. Follow-up studies . The traditional method of 
evaluating occupational programs and courses is to determine 
what happens to occupational graduates. If the graduates 
hold jobs or have gained entry to advanced training programs 
in the occupational area for which they were trained, the 
occupational program from which they were graduated can be 
considered to be effective. Although many factors having 
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little to do with the knowledge and skills that a program 
imparts affect the job that occupational graduates might 
obtain, follow-up data, when interpreted with caution, can 
provide useful information concerning important program 
outcomes • Follow-up surveys are expensive and difficult to 
execute. Many local education agencies — pairticularly 
secondary schools — have had difficulty or have declined to 
become involved with the time-consuming processes of following 
up occupational graduates. Problems typically characterizing 
most survey research also plague those who attempt a follow- 
up study: validity of responses; rate of returned question- 
naires; selectivity in who returns questionnaires; and so 
forth. Data analysis of follow-up results presents yet 
other difficulties . 

Despite the problems of obtaining and processing 
follow-up data, occupational educators continue to include 
and expand the role of follow-up data in their planning and 
assessment of program outcomes. Moreover, many educators 
would like to execute long-term or longitudinal follow-up 
studies to determine what happens to graduates two or more 
years after program completion. 

Because of the difficulties of executing 
follow-up studies and because of the sustained interest of 
the occupational education community in follow-up results, it 
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was decided that RESOE should be capable of executing both 
short-term and long-term follow-up surveys, and should also 
be capable of analyzing follow-up results. 

b. Other forms of program evaluation . The system 
must be able to support the execution of course and program 
evaluations which are based upon course or program outcomes 
other than follow-up returns. Such outcomes include various 
measures of student mastery, as well as noncognitive factors. 

2 . The system must be able to support program monitoring 
and instructional support processes . In recent years, there 
has been growing interest in the development of behavioral 
objectives for occupational courses or course modules. 
Once objectives have been established, teachers may determine 
whether students have mastered objectives by using criterion- 
referenced tests (such as skills check lists) to measure 
skill acquisition. When such information is obtained, 
summarized, and reported to teachers, to guidance personnel, 
and to the students themselves while the course is still in 
progress , the information may be used for flexible resource 



For example, a behavioral objective for the figure clerking 
might be as follows: when given a bundle of 100 cash slips 
fastened by a rubber band, a rubber thumb, and a 10-key 
adding machine, the student is able to obtain a tape total 
from the adding machine. A second tape will then be run 
for verification. One tape will be dated and attached to 
the bundle of sales slips. The entire operation is to be 
completed within five minutes (Huffman and Byers , 1971). 
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allocation, to help individualize instruction and to identify 
those students who require special assistance to master 
particular subject matter. 

This form of instructional support requires sophis- 
ticated computer software and data maintenance capabilities. 
Thus RESOE should be designed so that the computer require- 
ments / such as computer test scoring procedures, computer 
generated reports of student mastery, various managerial 
reports, longitudinal data base mainwenance, can be provided 
by the system. In this way local and regional education 
authorities wishing to implement objective-based instruction 
using instructional support processes can be assured of the 
requisite computer capability. 

3. RESOE must be capable of providing support for 
student certification procedures . Students who master all 
of the vocational skills, key vocational skills, or perhaps 
simply a percentage of the skills taught in a given occupa- 
tional course or program could receive a certificate of mastery 
There are many points of view regarding student certification 
in occupational programs, and there is more than sufficient 
interest in certification to include the execution of 
certification processes as a functional specification for RESOE 



A variety of specific models already exist for carrying out 
this form of instructional support. One such model has been 
developed jointly by RRI and the Guilderland Central Schools. 
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Since many occupational programs extend over more 
than one year, a certification system would require a 
multi-year student data base. RESOE must therefore be able 
to maintain such a data base. As each student completes an 
occupational course or course module, RESOE must be capable 
of adding student mastery data to the data base. 

The system must be capable of simultaneously and 
easily meeting different certification standards which may 
be required in different programs- In some programs, 
occupational educators might think it necessary for students 
to meet all program objectives. In other programs, some 
fraction of the total number of objectives might be considered 
sufficient for certification. In still other instances, it 
might be considered important for students to meet certain 
critical objectives plus a fraction of all other objectives. 

Thus RESOE must be capable of incorporating a 
variety of certification criteria into certification algorithms, 
which would operate on student mastery data (e,g,, criterion- 
referenced tests administered to students or skills check 
lists) « The system must be capable of reporting to occupational 
authorities the names of students who met the certification 
criteria, and the names of those who did not. The system 
must also be capable of reporting additional information, 
such as the specific objectives not yet attained by students 
who did not qualify for certification, 
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4 . RESOE must be capable of facilitating the work of 
guidance personnel > In addition to a variety of administrative 
support functions (such as retrieval of course results or 
individual records) RESOE must be capable of executing a 
variety of special studies and providing other forms of 
assistance in the guidance area. For example, the system 
should be able to support the execution of special studies 

to help establish empirical entry requirements for various 
occupational courses. The results of such studies may be 
used to counsel students, and to channel them into appropriate 
prevocational or other special programs when appropriate. 
Other examples include studies of student vocational interests 
and abilities, surveys of the labor requirements of regional 
employers, and analyses of which student groups have been 
served by particular courses and special services. 

5 . Program costing and cost-effectiveness analysis . 
Many occupational directors have indicated that program 
costing capability would help in program planning processes, 
budget preparation, and so forth. The costs which RESOE 
must eventually be able to process include teachers* com- 
pensation, other program-specific costs (such as equipment 
and supplies) , and also program nonspecific costs (such as 
general administrative expenses, plant maintenance, and debt 
service) . 
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Once program costs are available from the system, 
the system must be able to analyze the relationships between 
program costs and program effectiveness. The program outcomes 
which must be utilizable within the system for cost--eff ectiveness 
analysis include extent of student mastery of piogram-specif ic 
vocational skills, student follow-up data, and other quanti- 
fiable program outcomes. 

6 . RESOE must fulfill state and local reporting require - 
ments . Occupational administrators , guidance counselors , 
and teachers are currently besieged with reporting require- 
ments. These requirements emanate from the U. S. Office of 
Education, from the State Education Department, and from local 
and regional sources. Since the various reporting require- 
ments are frequently uncoordinated, information collection 
processes tend to be redundant, i.e., much of the information 
needed for the multiple reports is collected more than once. 

RESOE must be designed so that it can meet reporting 
requirements without requiring expensive or redundant efforts 
on the part of local administrators and their staffs. In 
addition to fulfilling federal and state reporting require- 
ments, RESOE must also be able to provide summary statistical 
data for the purpose of answering questions raised by local 
groups such as labor unions, parent groups, school boards,, 
and the various advisory bodies which help Boards of 
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Cooperative Educational Services and local education 
authorities to formulate policy for occupc^tional education. 



statistical analyses, and other reports to facilitate the 
dissemination of information about occupational education. 
Such reports are needed for newsletters, public presentations, 
or for the development of an annual State of Occupational 
Education Report. 

7 . RESOE must be able to support the execution of 
various administrative processes . RESOE must be capable of 
performing a variety of administrative data processing 
functions. Administrative procedures which RESOE should be 
capable of supporting include test scoring, enrollment 
processing, scheduling , grade reporting and the generation 
of transcripts. The system should be capable of supporting 
such administrative processes because the data required for 
the execution of these processes is often the same data 
that are needed to meet the other functional specifications 
which have been defined for the system. By including some 
administrative processes within the system, it is possible 
to arrange administrative schedules so that data gathered 
once can be used for many different purposes. 
B. General System Design Concepts 

Figure 1 illustrates general design concepts for a 
system which will meet the functional specifications 



The system should also be able to provide suimnaries. 
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described in the previous section. The design concepts 
include a central facility which provides a variety of 
support to local and regional education agencies around the 
state. The central facility has the responsibility of 
fulfilling repetitive, standard reporting requirements 
emanating from federal, state and local sources, as well as 
the responsibility of coordinating and meeting the more 
individualized or nonstandard reporting and evaluation 
requirement emanating from these agencies. In order to 
meet both the standard as well as the nonstandard reporting 
and evaluation requirements of state, regional and local 
education agencies (not all of which are predictable in 
advance) , RESOE requires a comprehensive computer data base 
maintained by a highly flexible software system. These 
design concepts are further elaborated in the following 
paragraphs . 

1. A central facility . As indicated in Figure 1, a 
central data processing and control facility provides a 
variety of services to the State Education Department, as 
well as to regional and local education agencies throughout 
the state. There are a number of reasons for incorporating 
a central facility into the system design. 

First, if data are centrally located, the central 
facility can easily serve the needs of a variety of education 
agencies without causing data collection problems. When SED 
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needs a summary report about some aspect of occupational 
education, it can obtain the report it requires from a single 
information source. A central facility can also provide 
quality service to local and regional education agencies 
that are not geographically close to the central facility. 
There are many examples in the computer field of centralized 
service being supplied cost-effectively to remote locations. 

Second, by centralizing the information facility, 
economies of scale are obtained. It is possible to amass 
sufficient work from local, regional and state education 
agencies so that the hardware, software and personnel required 
to do the job properly are efficiently utilized, and con- 
sequently the costs for participating education agencies are 
attractively low. 

2 . Design concepts for the_generation of basic reports 
and analyses . A basic report or analysis is one which is 
needed repeatedly at regular time intervals. For example, 
the State Education Department annually requires statewide 
information on occupational enrollments and completions. 

The central facility has the responsibility of 
establishing guidelines and procedures for meeting the 
State Education Department's repetitive information requests 
with respect to occupational education. Data collection 
forms, instruction booklets and other necessary materials 
are provided by the central facility to regional and local 

4 
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education agencies. The procedures established by the central 
facility are not inflexible, but make ample allowance for the 
differences among agei7cies to which services are provided. 
Local and regional education agencies provide the necessary 
raw data to the central facility so that the State Education 
Department's repetitive and most of its nonrepetitive requests 
can be easily and cheaply met. The central facility 
establishes a core data base associated with each local and 
regional education agency that is providing occupational 
education. The core data base is then used to meet reporting 
requirements . 

3. Design concepts for fulfilling nonrepetitive requests 
for reports and evaluations . Nonrepetitive requests from 

the State Education Department and from regional and local 
education agencies are also met by the central facility. 
The central facility fulfills nonrepetitive requests by 
using the core data base if it contained the needed data. 
When additional raw data are needed, it can be provided by 
the local or regional education agencies , and added to those 
agencies' core data files. These supplemental data are held 
by the central facility either permanently or temporarily, 
depending upon the needs and desires of the education agency 
making the requests. 

4 . Management of the central facility's data base by 
SEED . A frequent obstruction in the development and 
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implementation of management information systems is the 
substantial cost associated with developing and testing 
computer software. When a system cannot be completely 
specified at the outset, but must evolve over time, software 
development costs can be devastating. Software developed on 
an ad hoc basis to fulfill initial specifications often has 
to be rewritten many times as the system specifications 
change and expand with system use. 

The management information system needed by 
occupational education must be capable of accommodating 
changes in specific system processes as occupational education 
undergoes further growth an : development, and as the 
educational norms for operationalizing concepts such as 
accountability and evaluation in education change. If the 
Reporting and Evaluation System for Occupational Education 
(RESOE) is to fulfill its purposes, it must be able to easily 
assimilate these changes without major revisions in computer 
software. Flexible software must be used which will not 
require major modification as system requirements change and 
expand. However, such software will be less efficient than 
software specifically written to fulfill a particular purpose 
(i.e., more computer time will be required). Highly 
efficient software is appropriate when specific system 
specifications are likely to remain stable for many years, 
and only a minimal number of nonrepetitive system applications 
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are- to be carried out. For RESOE, the costs saved by 
applying efficient software at any point in time would be 
offset several times over by the need for continual software 
redevelopment. Therefore, a critical design concept of 
RESOE is its utilization of highly flexible softv/are that 
would not require continual redevelopment. (A method is 
available to integrate flexible with production software 
for repetitive operations: See footnote p. 20.) 

A softv/are system called SEED (System for the 
Evaluation of Educational Data) meets RESOE *s requirements. 
SEED is a flexible software system designed for the maintenance 
and analysis of educational data. SEED is capable of main- 
taining the core and supplementary data that RESOE will 
require for its operations. SEED includes a full computer 
file storage and maintenance capability. Flexible data 
entry and checking, the merging of new information into 
pre-established files, and the capacity to handle missing 
data are among SEED'S data maintenance attributes. 

SEED is capable of summarizing and reporting the 
data it maintains in virtually any manner. Reports generated 



SEED is a software package developed by Riverside Research 
Institute, and by Mr. Roald Buhler of Princeton University . 
During the first year of RRI's work for the Office of 
Occupational Education, SEED was found to meet many of the 
software requirements of RESOE. Therefore, RRI assigned 
SEED to the State Education Department for utilization in 
educational applications . 
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by SEED may be in the form of frequency distributions, cross- 
tabulations, graphs of various types, measures of central 
tendency (mean, median, mode) , or simply lists of data base 
entries (e.g., students or personnel) selected on the basis 
of some attribute or combination of attributes included in 
the data base (e.g., educational background, ethnicity, 
enrollment in a given program, eligibility for special serv- 
ices, age) . 

Since data in SEED-generated reports are labeled by 
SEED in accordance with the directions of the educational 
agency requesting the report, the reports are readily 
intelligible to their recipients. 

In addition to SEED'S capacity to meet reporting 
requirements, SEED's analytic and data summarization capa- 
bilities are more than sufficient for the fulfillment of 
RESOE's other functional specifications. SEED is capable of 
processing and summarizing follov;-up data. Furthermore, 
grades, test results, attitude data, measures of student 
mastery and other types of data relevant to the evaluation 
and monitoring of occupational programs can be maintained 
and processed by SEED. 

SEED may also be used by RESOE to maintain a student 
certification system for occupational education. SEED is able 
to maintain the quantitative student mastery data on which 
certification would be based, and would also be able to 
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execute the various certification algorithms developed for 
various occupational programs, courses, and course modules. 

For purposes of special studies, program evaluation, 
and so on, SEED includes a full complement of descriptive 
and inferential statistical capability. These include 
analysis of variance and covariance, t-tests (for correlated 
and independent samples), various correlations (biserial, 
Pearson product moment, tetrachoric) , multiple regression 
analysis (standard and stepwise) , multiple discriminant 
analysis, factor analysis, and many contingency table 
analyses (including chi-square analysis and analysis of 
uncertainty) , and other measures of association. 

SEED is open-ended. The central facility or other 
user may attach subprograms, either permanently or temporarily, 
to SEED, so that processes which could not be executed by 
the standard SEED package may still be carried out. Thus if 
a request comes to the central facility that SEED cannot 
fulfill, it is possible to meet that request without removing 
data from the SEED-maintained data base. 

Finally, SEED can be used by persons without any 
previous training in computer science. It has an English 
language control system. Thus it would be possible for 
occupational educators to have direct access to their own 
data base over inexpensive remote terminals. 
* 

Production software for highly repetitive operations 
may also be attached. 
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5. Summary . This section has presented some of the 
major design concepts of RESOE: The concept of a centralized 
facility operating under SED auspices; the concept of 
establishing a core data base for processing repetitive 
requests from the State Education Department and other 
education agencies; concepts for processing nonrepetitive 
requests; and the computer software concepts for data 
maintenance and processing. The next sections describe how 
the design concepts have been operationalized. Specific 
procedures are described for the implementation of RESOE in 
several regions of New York State. 
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III. General Strategy for the Implementation of 
RESOE in New York State 
This section describes the general approach to the 
implementation of RESOE, which has been developed with and 
approved by the State Education Department's Office of 
Occupational Education. A more detailed discussion follows 
in the succeeding section of how the implementation strategy 
was applied during this past year in three occupational 
planning regions in New York State. 

Implementation of RESOE occurs in two logically distinct 
but overlapping stages. In the first stage, the core or basic 
system which is needed to fulfill repetitive basic reporting 
requirements is established. The repetitive reporting require- 
ments which have been emphasized are those emanating from the 
U. S. Office of Education and the New York State Education 
Department. Local and regional education agencies have often 
had difficulty .providing valid data to fulfill these require- 
ments in the past . 

The second stage of implementation involves the utiliza- 
tion of RESOE by local, regional, and state education 
agencies for nonrepetitive report requests, and for locally 
or regionally based processes such as program evaluation, 
costing, monitoring and instructional support. In practice, 
the two stages frequently overlap, although they are logically 
distinct . 
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The following paragraphs describe the implementation 
strategy for Stage I. Stage II implementation is far more 
flexible and largely dependent upon the Or.rticular functions 
for which various education agencies decide to obtain RESOE 
support. 

Figure 2 is a flow diagram which illustrates the 
sequence of major steps in the implementation of Stage 
The process begins with the collection of student enrollments, 

•k 

some personnel data , and some information about program 
offerings. The student enrollment data minimally include 
the following information on all students enrolled in occu- 
pational courses: Social Security number, demographics, 
type of student, grade level, birth date, and codes for the 
occupational courses in which the students are enrolled. 
Student name, address and some other information (e.g., purpose 
for taking the course) are also collected at certain sites. 
(The specific procedures for collecting these enrollment data 
are reviewed in sections IV and V of this report.) 



The personnel data collected in phase I are only those data 
which are needed to fulfill repetitive reporting requirements, 
and which are not in the State Education Department's Basic 
Educational Data System (BEDS) . No attempt is made in Stage I 
implementation to develop and maintain a comprehensive person- 
nel file. 



F/242-5-00 



- 23 - 




Student 
enrol I.Tient 
data 
^collected i 



Personnel 
data 
Vcollected > 



\ 



Program 
data 

collected 



D<itJi prepared 
for cntr> 
VSEEO data base/ 





Develop Student 
Pile Cor enroHces, 
Personnel rile, and 
»>?09ran File 




StEO afjqiregates 
enrol Inont data 
and personnel 
data m accordance 
with ir.andated 
reqjuireronts 















Are 
courses 
started a:t 
iMtial 
^nrol Iment »* 

\ Collect 7 
\ enrol Inonts / 
\ for new / 
\courses / 

Prepare data 7 
for entry into / 
SEED £ ^^se/ 



Update student 
file of 
enrol lees 



Store new 
enrol Inent 
data 



SEED aggregates 
enrollrxjn t 
data for 
annual reports 

1 



Annual vnroll- 
nent report for 
educational 
agencies 



STUDENT ENROLLMENT 



S'UOENT ENROLLMENT 
DATA UPDATE 



Q F/242-5-00 

ERIC 



of ;j. eii. I 



Send out 




tollow-up 




survey to 


A 


forner 




students 





STUDENT FOLLOW-UP 



Octcrninc which 
students did not 
return their 
surveys 

i 



Send d second 
Ri'^.iling 



/enter returned 
/surveys into , 
I s,udent fiie j 




Prepare report 
fox SED 



I Follow-up report 

for SED -and 
I education 
agencies 




\Z^A obtains 7 
follov-up Inforn*-/ 
tion on forms / 




ta entered Into / 
udent file / 



Prepare LEA updated 
follow-up report 
for agency 

i 



rol low-up report 
for agoncy 




Fig. 2. Flow diagram illustrating 
the sequence of steps in the first 
implementation stage of the Report- 
ing and Evaluation System 
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The enrollment data are sent to the central facility 

where they are prepared for entry into the SEED"-maintained 

data base. The enrollment data are then placed in a core 

student file which is created for each participating local 

education agency. In late October, the SEED-maintained 

student enrollment files are temporarily closed, and SEED 

is used to aggregate the student enrollment and personnel 

data in accordance with the reporting requirements mandated 

under the Vocational Education Act of 1963 and the Vocational 

* 

Education Amendments of 1968. The required enrollment and 
personnel reports are then sent by the central facility to 
the State Education Department. Copies of these reports are 
also prepared for participating regional and local education 
agencies . 

Many education agencies have courses which begin after 
the initial enrollment data are collected. This situation 
is particularly characteristic of adult occupational courses, 
which tend to begin at several points during the academic 



A report of follow-up studies of the previous year's 
graduates is also required at this time of year* Since the 
previous year's graduates are not in the SEED-maintained data 
base in the first year of RESOE implementation, it is not 
possible for RESOE to fully execute the required follow-up 
study. However, PESOE is used to ease the burden on local 
and regional education agencies; follow-up data are obtained 
from the education agencies, aggregated and reported to the 
State Education Department in accordance with state-prescribed 
formats . 
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year, or are repeated several times between September and 
June. Enrollment data are collected and forwarded by the 
education agencies throughout the year. These additional 
enrollment data are prepared for entry into the data base 
and are then added to the basic student files that were 
initially constituted in the fall. In the spring, the student 
files are used to provide enrollment data which the education 
agencies need for their annual report to the State Education 
Department. 

In mid-April, a list is developed of occupational educa- 
tion graduates and drop outs who are to be followed up. (The 
particular procedures for developing this list are reviewed 
in section III of this report.) Most student information is 
already in the basic student enrollment file. A follow-up 
student file is created by RESOE, with much of the information 
in the follow-up file taken from the pre-established enroll- 
ment file. 

During the following fall, the follow-up file is used to 
execute a follow-up survey. (Details of the follow-up survey 



* RESOE does not file the annual report on behalf of the 
education agencies, because RESOE does not have the information 
at this stage of implementation to complete many of the annual 
report sections. 

** At about the same time, the student enrollment data are 
once again being placed into student files by the central 
facility, the new enrollment data are being compared with the 
previous enrollment data , and a longitudinal or long-term 
student data base is created by SEED. This process is 
repeated every year. 
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questionnaire and procedures are discussed in later sections 

of this report*) As the follow-up surveys are returned by the 

graduates, they are prepared for computer entry, and then 

entered into pre-established follow-up files. 

If a second survey questionnaire is to be sent to those 

students who did not respond to the first questionnaire, SEED 

checks the follov/-up files to identify those students who did 

not return their first questionnaire. A second questionnaire 

* 

may then be sent. The surveys returned from the second 
questionnaire are prepared for entry, and entered into the 
SERD-maintained follow-up file. 

One week before the State Education Department's final 
deadline for the follow-up report, the follow-up file is 
temporarily closed and the central facility applies SEED 



* The number of returns obtained from second follow-up 
mailings are usually not sufficient to justify the costs of 
conducting a second mailing. However, some education agencies 
require nearly complete data and will plan to personally 
follow-up (through phone calls and field trips) as many 
graduates as possible. A second mailing may sometimes help 
to reduce the burden associated with the personal follow-up. 
SEED'S analytic capacity may be used to determine whether a 
second mailing or personal follow-up is warranted. If the 
sample of students who returned their questionnaire approxi- 
mates, in important respects, the population of students who 
are being fol?.owed up, then the follow-up survey may be con- 
cluded, and SEED'S inferential statistics may be used to 
draw conclusions about the population of graduates from the 
sample of graduates who responded. 
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to the aggregation of follow-up data and to the generation of 
the SED-prescribed follow-up report. Graduates who have not 
returned their surveys before the files are closed for aggre- 
gation are counted as "status unknown" on the state report. 
Participating local and regional education agencies receive 
copies of this report. After the follow-up report is submitted 
to BED, the follow-up file is reopened for the entry of any 
additional questionnaires which may have been returned. 

Some of the participating local or regional education 
agencies may wish to personally follow-up (through telephone 
calls or site visits) those graduates who did not return their 
questionnaires. These education agencies are provided with a 
precoded form for each student who has not provided follow-up 
data. This form is filled out by the agency's own staff when 
the student is contacted. When the agency believes tha^ 
it has a sufficient number of returns for its purposes, the 
agency sends the follow-up forms to the centra], facility 
where they are prepared and entered into the pre-established 
follow-up file. The follow-up file is then used to conduct 



This follow-up procedure involves two data aggregations. 
The first aggregation occurs in time to meet the State Educa- 
tion Department report deadline. This deadline is determined 
bv the U. S. Office of Education reporting schedules. The 
second aggregation occurs aftei. the education agency decides 
that it has completed its follow-up study. While this pro- 
cedure requires two data aggregations, it avoids the burden 
of conducting two relatively independent follow-up studies: 
one to meet mandated reporting requirements, and the other 
for local purposes. 
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any anaxyses on follow-up data that are requested by the 
education agency. The student follow-up file is permanently 
maintained by the central facility in the event that education 
agencies wish to carry out follow-up surveys for graduates 
who have been in the labor force for two or more years . 

The enrollment process coupled with the follow-up 
process serves to meet a variety of reporLing requirements 
as well as to establish SESD-maintained basic student files 
which can be used without modification for a variety of 
purposes. These files can be supplemented with additional 
information so that still other processes may be carried out. 
The specific procedures for establishing the basic student 
files in various regions of the state are described in the 
following -section . 
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IV. stage I of RESOE Implementation 
in Planning Regions 9 and 1 

A. Implementation Sites 

During this past year, RRI implemented RESOE in two 
planning regions selected by the State Education Department's 
Office of Occupational Education. The selected planning 
regions were planning regions 9 and 1.* The implementation 
sites in planning region 9 were: 

• The city of Syracuse; 

• the Onondaga County BOCES and its member school 
districts; 

• the Oswego County BOCES and its member school 
districts; 

• the Cortland-Madison Counties BOCES and its 
member school districts; 

• the Cayuga County BOCES and its member school 
districts; and 

• the Tompkins-Seneca-Tioga Counties BOCES and its 
member school districts. 



These planning regions are referred to in RRI ' s quarterly 
progress letters as planning regions 12 and 2 respectively, 
which were their previous numeric designations. Former planning 
region 12 is geographically identical to planning region 9. 
Former planning region 2 included only Nassau County, whereas 
planning region 1 includes both Nassau and Suffolk counties. 
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The implementation sites in planning region 1 were: 
• The Nassau County BOCES; and 

« the Nassau County Vocational Education ExtGnsion 
Board (VEEB) . 

The specific procedures for implementing RESOE in these 
planning regions are described in the following paragraphs. 
B. Procedures for the Establishment of the Basic Student 

Enrollment File 

As indicated in Figure 2, the objective of collectincj 
student enrollment information is the establishment of basic 
studont files for each participating education agency- These 
files serve to fulfill mandated reporting requirements and a 
variety of other purposes. The procedures under which a 
SEED-maintained basic student file was established may be 
logically separated into three parts: the dissemination of 
information and materials to the participating education 
agencies; the collection of the student data; and the data 
processing required to enter the student data to establish 
the files and to generate enrollment reports. 

1 . The dissemination of materials and information . RRI 
worked directly v;ith an occupational coordinator at each of 



The system was partially implemented in Nassau County school 
districts for the purpose of meeting State Education Depart- 
ment reporting requirements, but a permanent SEED-maintained 
data base was not escablished for the districts' occupational 
students. 
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the six BOCES included in the RESOE implementation. These 
coordinators were either BOCES occupational directors or 
individuals designated by them* RRI also worked directly with 
the occupational director at the City of Syracuse School 
System. With the exception of the Syracuse school system, 
the BOCES coordinators were responsible for coordinating the 
implementation of RESOE at the BOCES occupational centers, and 
also at BOCES' member school districts providing their own 
occupational programs. 

In planning region 9, a number of meetings took 
place betv/een RRI and the regional coordinators in which the 
concepts and procedures of RESOE were explained. Three of the 
five BOCES coordinators in region 9 accepted RRI's offer to 
make a joint presentation to school district principals and 
guidance personnel concerning RESOE implementation in local 
school districts. Similar meetings were also held with the 
Nassau county coordinator and with the principal of one of 
Nassau's central high school districts. 

In late August and early September, sufficient 
quantities of materials were sent to the coordinators for 



* 

RRI worked directly with the Syracuse School System, because 
Syracuse is one of the big-six cities of New York State and is 
not connected with a regional BOCES. All other school districts 
in the two planning regions are associated with a BOCES - 
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data collection purposes. These materials are contained in 
Appendix 2 and are briefly described below* 

a. Student Survey Procedures for the Occupational 
Education Data System* This is a booklet which describes the 
general procedures to be followed in obtaining the information 
required for the basic student file. It contains some needed 
definitions (such as the definition of an occupational course) 
and instructions for how to provide information about occupa- 
tional courses on a form designed for this purpose. 

b. Program Classifications for Occupational 
Educcvbion. This booklet contains standard program codes for 
occupational education. These codes are to be entered on 
occupational course forms (described below) . 

c. Occupational Course Forms . These forms were 
used by administrators in occupational area centers and school 
districts to provide information on occupational courses. 

One form was to be used for each course. The form was pro- 
vided as a pressure sensitive label, so that it could be 
placed on the outside of an envelope which would subsequently 
contain student enrollment data for the course. The following 
essential information is obtained from this form: 

• Code for the educational agency which is 
providing the course 

Federal occupational education program code 
for the course 
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• Indication of whether the course is new or 
continued 

• Level of course: high school, college or adult 

• Type of course: regular, cooperative, or 
occupational special education 

• Name of teacher giving the course 

• Room number in which the course is given 

• Course title 

d. Student Course Enrollment Form. Instructions 



for Classroom Administrators . This booklet contains class- 
room instructions for the completion of student course enroll- 
ment forms. The booklet also contains guidelines for 
identifying handicapped and disadvantaged students, and for 
providing this information on each student's form. The 
booklet also suggests data checking procedures. 



is filled out by each student for every occupational course 
in which he is enrolled. The form requests information for 
the basic student file as well as other information needed by 
occupational administrators for enrollment and registration 
processing. The following information is requested of the 
student: 



Student Course Enrollment Form. This form 



♦ Name 



• Address 



• Telephone number 
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• Date of birth 

• Course code (provided by administrator) 

• Social Security number 

• Military status (veteran or nonveteran) 

• Demographic information (race and sex) 

• Information about cooperative work experience 
jobs 

• Grade level (for high school students) 

• College students indicate whether they are 
full time or part-time, first year students, 
or beyond the first year of study, and in 
or not in a degree program 

• Adult students indicate their main reason 
for taking the course: preparation for new 
job; job advancement; part of an apprentice- 
ship program; or for avocational reasons 

In addition to the above, the form also has 
room for coded information concerning type of student (regular, 
disadvantaged or handicapped), and whether or not the student's 
parents are migratory agricultural workers. 

f. Additional materials . Other materials were 
also sent to the coordinators for use in the collection of 
enrollment data. These materials (contained in appendix 2) 
included : 
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• Lists of numeric location codes for local 
and regional education agencies 

• Prototype of a principal's letter of explanation 
(to be used lor data collection in school 
districts) 

• A letter of introduction from Dr. Robert 
Seckendorf, Assistant Commissioner for 
Occupational Education, State Education 
Department 

• A supply of large brown envelopes 

• Some additional forms for providing data on 
teachers in occupational adult education , 
and on paraprof essionals in secondary 
education (see appendix B) 

2. Collection of the student enrollment data . The 
objectives of the collection process are: a) to obtain a 
completed enrollment form for each student in each course; and 
b) to obtain pertinent information about the courses in which 
the students are enrolled. In operational terms, this means 
that for each course, the central facility required an envelope 
with an occupational course form affixed to the outside and 
completed student course enrollment forms on the inside. 

Figure 3 illustrates the alternative procedures 
(developed by RRI and the coordinators) which were employed 
for collecting the student enrollment data. After orientation 
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Fig. 3. Flow diagram of RESOE's 
student enrollment procedures. 
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meetings were held, and materials were sent to coordinators, 
occupational administrators in each participating school 
and BOCES occupational area center prepared a list of occupa- 
tional courses offered in their education agency. They then 
completed an occupational course form (i.e., they put requested 
course information on a pressure sensitive label) , using the 
instructions provided in Student Survey Procedures for the 
Occupational Education Data System . 

The method by which the enrollment forms were 
administered to occupational students was chosen by the local 
school administrators. Three different methods were tried in 
1972, differing in the manner in which personnel and student 
resources were allocated to the enrollment survey. All were 
successful in collecting the required data. 

a. The classroom administrator method was the most 
widely applied procedure throughout both planning regions. 
The first step in this method was the labellinc and filling of 
the course envelopes. Each prepared envelope contained a 
sufficient nun±)er of student course enrollment forms and a copy 
of Instructions for Classroom Administrators . At certain loca- 
tions, the envelopes also contained a letter from the local 
principal or area center supervisor explaining the reasons 
for the survey. 



F/242-5-00 



- 38 - 



ERLC 



50 



The envelopes were delivered to the local 



classrooms. The label on the envelope (course form) facilitated 
delivery by providing the address of the classroom, local 
course title, and teacher's name. The envelopes were distrib- 
uted to most courses during the first week in October. (For 
courses not in session in October, the materials were distributed 
shortly after the course began.) 



completion of the student enrollment forms was supervised by 
the classroom teacher assigned to the course, or by a school 
official who collected the enrollment data from all occupa- 
tional courses. In most instances, the classroom teacher 
was selected. (In adult classes it was not uncommon for a 
school official to visit the classrooms and administer the 
enrol Iment forms . ) 



contained administration directions, and instructions to be 
read to students. Ten to fifteen minutes of class time were 
consumed in this method. The administrators' instructions 
also contain directions for checking the data entered by stu- 
dents, for identifying and coding student type (regular student, 
disadvantaged student, handicapped student) on each student's 
enrollment form, and for returning the completed material to 
an administrator designated to receive them. 



Under the classroom administration method. 



The administrators* manual inside the envelope 



b. 



The student assembly method of collecting the 
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student enrollment data was applied at the Cortland-Madison 
BOCES. In this method, occupational students meet in the same 
room so that all students in all courses can complete enroll- 
ment forms simultaneously in a single administration session. 
A school official administers the forms just as in the class- 
room administration method. The students are requested to 
enter on their forms the Office of Education Program Code 
niomber for their occupational course. They obtain this code 
number from a list of courses and corresponding numbers which 
is posted on a blackboard. 

After the students have completed the forms, 
the school official checks them and enters student type (regular, 
disadvantaged, handicapped) . The completed student enrollment 
forms are then sorted into groups according to occupational 
program code. Each group of forms is placed in an envelope, 
and a course label is prepared and attached to the envelope. 
At this point, the enrollment data from the student assembly 
method and the classroom administrator method are in the same 
form. 

c. The coordination of the RESOE enrollment 
survey with traditional registration . Perhaps the least time 
consumptive procedure for obtaining kESOE enrollment informa- 
tion is to coordinate the enrollment survey with course 
registration. This method was successfully used at the Cayuga 
BOCES. Occupational students completed the RESOE Student Course 
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Enrollment Form at the time that they registered for BOCES 
courses and also whenever they changed courses. As in the 
other administrative methods, the student's classification 
(regular, disadvantaged or handicapped) was added after the 
student completed his part of the form. As in the assembly 
method, the forms were eventually sorted into course groups. 
Each group was placed in an envelope and labeled with a 
course form. 

The coordination of registration and the RESOE 
enrollment survey saves student and personnel time because 
much of the information required for the student file must 
be obtained at registration time for local administrative 
processes. If the Course Enrollment Forms are completed at 
registration time, it is possible to use the information on 
the forms for a variety of local administrative purposes before 
the forms are grouped by courses, and sent to the central 
facility. For example, the Cayuga BOCES used the completed 
student course enrollment forms as source documents for the 
production of class rosters to be used in the Cayuga attendance 
system. The forms were also placed in student files, where 
they served as a convenient and easily updated source of 
student information. 

3 . Data processing required for the establishment of 
the student file . The coordinators sent the enrollment 
envelopes to RRI, (which served as the central facility) 
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Inhere they were checked in, and keypunched. A SEED-maintained 
student file was then established and enrollment report sum- 
manes (see Figure 3) were produced. 

4 . Keeping the enrollment files current . Throughout 
the school year, enrollment information was obtained for 
courses which were begun after the initial enrollment data 

were sent to the central facility. The procedures for collecting 
this supplementary enrollment information are similar to those 
used in the initial enrollment collection process. 
C. Updating the Student File 

In the late spring, RESOE's student files are changed 
as needed and some new information is added. With respect 
to changes, virtually any of the data in the basic student 
file that are found to be incorrect may be changed, including 
new surnames for recently married female students. 

The student file also can be .updated at this time with 
address and telephone changes for those students who are to 
be followed up in the annual follow-up survey. Students to 
be followed up include those who have previously taken or are 



* 

On behalf of the education agencies participating in RESOE 
implementation, RRI also received forms containing personnel 
information and forms (OEP-2) containing follow-up results of 
the previous year's graduates. The data on these forms were 
keypunched. SEED was used to aggregate the personnel and 
follow-up data, and to provide the required follow-up and 
personnel reports to the State Education Department. 
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currently enrolled in an occupational program and are graduating 
from high school at the end of the current semester, those who 
have dropped out during the academic year, adult students who 
have completed their occupational p^-eparation for a new job; 
and, in some instances, students who are transferring to 
another educational agency. 

The RESOE spring student file update processes are 
illustrated in Figure 4 (which continues the RESOE flow chart 
begun in Figure 3). As in the enrollment procedure, procedural 
particulars vary with the type of educational setting, the 
preferences of the occupational directors, local resources, 
and local capabilities. Three alternative update options 
are illustrated in Figure 3, and are bi'iefly described in the 
following paragraphs. 

1 . Updating the Course Enrollment Forms (method a) . 
In this method, the original enrollment forms are returned to 
the local or regional education agency for redistribution. 
The redistribution of these forms to students follows the same 
procedure that was selected by the education agency for the 
initial distribution. Blank forms are distributed to students 
who were not previously included in the enrollment survey. The 
students update their forms by filling out a "Student Course 
Completion" section (see appendix 2) . This section asks the 
student to indicate any changes in the information (address , 
telephone number, etc.) provided earlier in the school year. 
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The local education agency identifies in the "Student Course 
Completion" sections, any students who have dropped out. The 
forms are then returned to the central facility for processing. 

2 . Updating files for graduates and dropouts (method b) . 
Under this option, the regional coordinator sends blank infor- 
mation cards and a SEED-generated list of all enrolled students 
to school districts and other agencies where occupational edu- 
cation is offered. The cards are distributed to graduating 
students who complete them with the following information: 
name, address, phone number, date of birth, sex, and Social 
Security number. After the cards are completed, they are 
returned to the regional coordinator. The names of student 
drop outs are circled on the enrollment list, and this list is 
also returned to the coordinator. The coordinator sends all 
materials to the central facility. 

3 . Method of coordination with a local automated record 
system (method c) . A third option for the RESOE spring student 
file update is for local and regional education agencies to 
provide, from an automated record system, a computer tape or 
punched cards with the necessary information. The tape or 
cardt> would include current addresses for graduating students 
who took occupational courses and a unique student identifier, 
such as Social Security number, that was already included in 
the RESOE enrollment file. 

The spring update serves a variety of purposes. 
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First, it enables the central facility to maintain up-to-date 
information. Second, it provides a vehicle for occupational 
educators to add additional end-of-year information such as 
students* grades, or placement data to the data base* 
Finally, it serves to identify the students to be followed 
up in the subsequent follow-up survey. 
D . The Follow-up Survey 

The follow-up survey of occupational education students 
is conducted annually by RESOE. Execution of the survey begins 
in early September, and continues through October and November. 
This section of the report describes the follow-up procedures 
which are scheduled for implementation in planning regions 
1 and 9 in 1973. 

1. Who is to be surveyed in the follow-up survey . 
In order to fulfill state reporting requirements, the following 
groups must be included in the follow-up survey: 

• Recently graduated secondary students who have 
taken occupational courses 

o Secondary students who have dropped out of school 
during the previous academic year, and who have 
completed an occupational program, or were enrolled 
in an occupational program at the time they dropped 
out 

• Adult students who were enrolled in occupational 
courses to prepare for new jobs 

• Postsecondary graduates 
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At the option of regional and local administrators. 



additional student groups may also be included in the follow- 
up survey. For example, in planning regions 1 and 9, many 
education agencies will include secondary occupational graduates 
of previous years, because follow-up data from graduates who 
have been in the labor force for a year or two is a more valid 
indicator of program effectiveness than the follow-up results 
of more recently graduated students. Other groups which local 
or regional education agencies might select for inclusion in 
the RESOE follov7-up survey are adults who took occupational 
education to supplement their skills and knowledge for a job 
they already have, and adults who participate in occupational 
courses for avocational purposes. 

2. The basic follow-up questionnaire . The basic follow- 
up survey instrument consists of four basic questions which 
are indicated below. 



a . 



If you are continuing your education, please 



darken in ( J|^) the box (/ /) or boxes to 



indicate where you are currently enrolled. 



/ / 



Daytime high school 



/ / 



Board of Cooperative Education Services 
(BOCES) 



n 



Adult Continuing Education 



/ / 



Private Occupational School 



/ / 



On-the-job training where you work 
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/~ Apprentice program where you work 
/~ / Occupational major at two-year college 
/~ Liberal arts major at two-year college 
/"7 Other major at two-year college 
/~ Four-year college 

/ / Other (please write in) 

Are you employed at this time? 

/"7 Yes, employed full time 

Name of Employer: 

Job Title: 

/~ J Yes, employed part-time 

Name of Employer: 

Job Title: 



/ / No, but looking for a job 

(Please contact our Placement Office 

if you would like help in finding a job) 

/~ No, and not looking for a job at 
this time 
What is your gross weekly income? 

$ 

At your job, do you make use of the skills and 
information that you learned in your occupa - 
tional courses? 

/~ The skills and information are very 

important in my job 
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/~J The skills and information are not 
as important now as when I started 
my job 

/~ The skills and information are not 
very important in my job 

/~ J The courses only covered half of 
what I really need to know 

rj 1 do not need the courses I had for 
the job that I have now 
e. Are you on active duty in the Armed Services? 

rj Yes 

rj No 

The first question provides information about students 
who are continuing their education- The second question permits 
the student to indicate whether he has a job. The employer's 
name and job title are included so that placement counselors 
may know where former students are employed and thus wh^re 
future students may find employment. The inclusion of the 
student's specific job title also serves another purpose, A 
numeric code from the Dictionary of Occupational Titles (DOT) 
may be assigned by the central facility or local education 
agency to each student's job. It v/ould then be possible to 
determine quantitatively the extent of relatedness between 
the entry-level skills required for a student's job and the 
entry-level skills required for the cluster of jobs for which 
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the student was trained in occupational education. 

The third question asks students to indicate gross 
compensation for their work. The fourth question solicits 
the students* opinions of the relevance of their occupational 
education to their jobs, and the fifth question is included 
to determine military status. 

Questions one, two, four and five are needed to 
fulfill mandated reporting requirements. Question three has 
been included because it nearly always appears in some form 
on the follow-up surveys of local and regional education 
agencies. Ansv;ers to this question permit determination of 
the benefits side of program costs-benefits ratios and also 
serves as a partial validation check on the job title pro- 
vided by the student in answer to question two. 

3 . The additional or optional follow-up questionnaire . 
Many of the local and regional occupational education agencies 
in planning regions 1 and 9 have been carrying out their own 
follow-up studies in the past, and have used follow-up results 
to assess various aspects of their occupational programs. 



The Coordination of Program Planning and Evaluation Systems 
For Occupational Education, Voliame II: A Targeting System for 
Occupational Educ^'ion describes procedures for quantifying 
the degree of relationship between students* jobs and their 
occupational training . 



F/242-5-00 



- 50 - 



For example, one agency is engaged in longitudinal follow- 
up studies which include occupational graduates as well as 
"control" students who were not enrolled in occupational 
courses . 

ThuS/ certain local and regional educational agencies 
have formulated specific questions which they would like to 
ask their graduates in the yearly follow-up survey. Many other 
agencies have identified various occupational programs and 
services about which they would like some feedback from their 
former students. 

Follow-up survey instruments generated by local and 
regional occupational agencies frequently include questions 
which allow the agencies to assess the effectiveness of their 
guidance and placement services. Questions concerning job 
satisfaction and job mobility are also included. In addition, 
students are often asked to assess the relevance of various 
components of their previous occupational and regular education 
to their role in the world of work. 

RESOE meets the individual follow-up requirements 
of local and regional education agencies by adding each agency's 
follow-up questions to the basic follow-up questionnaire. It 
is thus possible for RESOE to simultaneously meet mandated 
reporting requirements and fulfill "clie needs of individual 
education agencies without causing duplicated work efforts. 
The procedures for conducting the follow-up survey are dis- 
cussed in the following section. 
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4 . RESOE procedures for conducting the follow-up 
survey . RESOE 's procedures for conducting the follow-up 
survey are illustrated in Figure 5. The figure contains a 
flow diagram which continues from the sp.ving update illustrated 
in Figure 4, and ends with the preparation of final follow-up 
reports to participating local education agencies. 

As indicated previously, one of the purposes of the 
spring update procedure is to determine which students are 
to be surveyed in the next follow-up study. This information 
is obtained through one of the alternative updating procedures 
described earlier in this report (seo Figure 4) . The central 
facility then uses SEED to create a follow-up student file 
for each local and regional education agency participating in 
RESOE. The minimal information contained in this file is the 
data provided on each student at enrollment time as well as 
the student's updated address, telephone number, etc. The 
file is used to generate several mailing labels for each 
student to be followed up. The mailing labels include the 
student's name and address as well as a unique numeric temporary 
identifier (not Social Security number) which enables SEED to 
merge the student's returned follow-up questionnaire data into 
the follow-up file. 

During the spring, the central facility works with 
the BOCES coordinators to formulate and gather questions for 
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Fig. 5. Flow diagram of RESOE 
follow-up procedures 
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the optional follow-up survey. These questions are then 
organized by the central facility for coordination with the 
basic follow-up questionnaire. 

In mid-August, the follow-up questionnaires are 
sent out by the central facility. A label is attached to p.ach 
student's questionnaire. This label appears through an open 
window envelope so that it serves as a mailing address for 
the Postal Service. A postage-paid business reply envelope 
is included for the student to return his completed question- 
naire. 

As the questionnaires are returned Lu the central 
facility, they are entered into the SEED-maintained follow-up 
files that were created during the previous spring- In October, 
SEED is used to determine which students have not yet returned 
their questionnaire, and to generate some statistics on the 
rate of return and sample bias in the returned data. 

The central facility an^ .the regional coordinators 
then decide whether a second centralized mailing should be sent 
to students who did not return their questionnaires. If a 



After a few years of RESOE operation, it will be possible to 
establish a bank of optional follow-up questions which would be 
referenced by categories such as job satisfaction, guidance 
activity, placement activity, assessment of training relevance, 
job mobility and so forth. The bank would also contain relevant 
empirical data (from prior utilization) on the efficacy of the 
questions, and data which may be used by education agencies to 
compare their follow-up results to "norms" based upon the data 
obtained from other regions. 
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second mailing is carried out, it would probably contain just 
the basic questionnaire. The re.sults of the second question- 
naiiG are also entered into the central facilitxis data base 
as they are returned. 

At a determined cut-off date, the file is temporarily 
closed and SEED is used by the central facility to fulfill the 
mandated follow-up reporting requirements for the State Educa- 
tion Department. Students V7ho have not responded are listed as 
"status unknown" on the state report. At the same time, the 
central facility alsc uses SEED to generate a report of the 
extent to which the group of students who returned follow-up 
data constitute an unbiased sample of the ent're population 
of students who were included in the follow-up study. 

The return rate and sample bias report are then 
used by each education agency to decide whether it wishes to 
allocate the resources required to "personally" follow up those 
students who have not responded through the mail. If the local 
or regional education agency does not wish to expend personnel 
and other resources in this manner, the agency submits a 
request to the central facility for any additional final reports 
or inferential statistical analyses it would like to have 
carried out on the data from the basic and optional follow-up 



* 

Second mailings rarely increase the rate of follow-up more than 
five to ten percent and are thus usually not worth executing. 
Second mailings would only be justifiable under relatively 
unusual conditions when there was sufficient reason to project 
a meaningful return rate. 
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questionnaires. Any essay-type optional questions which 
were inserted by local agencies are returned to them for 
review and local coding operations. 



have not returned their follow-up forms, the central facility 
provides lists of these students, along with forms and guide- 
lines for collecting follow-up information. The local educa- 
tion agency then completes a follow-up form on each st' dent 



for whom data are available within the agency. The agency 
may collect additional follow-up data by visits (to parents, 
employers, etc.) or through a telephone survey. The forms 
are returned to the central facility and the follow-up data 
are added to the SEED-maintained follow-up data base. Final 
follow-up reports and analyses are then produced by the central 
facility at the agency's request. A final report of any sample 
bias that still remains in the data is also produced. 



education agency and the generation of the agency's final 
follow-up reports is dependent upon the amount of time that 
the agency wishes to expend on the follow-up project and when 
they are able to expend it. However, it is assumed that 
most agencies will complete their work no later than March of 
the following year. 

The occupational status of some students may be known to 
the placement office or teaching personnel. 



If the agency wishes to follow up students who 



* 



The timing of the "persona 



1" follow-up by the local 
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5 . Increasing the rate of return in the follow-up 
survey. There are several ways of reducing the work efforts 
and costs associated with the execution of the RESOE follow-up 
study. RESOE 's procedures may be made more efficient and 
cost-effective by reducing the need for second mailings and 
personalized follow-up canvassing by local and regional educa- 
tion agencies. This objective can be realized by implementing 
practices that will result in an increased rate of return for 
the initial central follow-up mailing. 

A practice which may help to increase the return 
rate is to inform students of the importance of the survey. 
As part of the spring update procedure, students may be told 
about the follow-up survey and asked to complete and return 
their questionnaire immediately after they receive it in the 
fall. 

Another means of increasing the rate of return to 
the initial follow-up mailing is by offering the services of 
the education agency * s placement office to former students 
who are looking for a job or are not satisfied with the job 
that they have. This offer of job placement motivates the 



* 

An efficient approach to the collection and analysis of 
certain types of follow-up data exists within the Targeting 
System for Occupational Education, developed by Riverside Re- 
search Institute. See The Coordination of Program Planning 
and Evaluation Systems for Occupational Education/ Volume II : 
A Targeting System for Occupational Education / 1972. 

** 

This technique has been successfully used by the Nassau 
BOCES . 
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student because there is a personal reason for him to complete 
the form, i.e., he is not just helping to plan for future 
generations of occupational graduates. Moreover, many 
occupational education ag^.ncies would like their former 
students to make use of their placement services, and the 
follow-up instrument can thus serve as a reminder to students 
of the availability of job placement support. 

Still another means of increasing the rate of return 
to the initial follow-up mailing is by carrying out a sweep- 
stakes as part of the survey process. All former students 
who returned completed follow-up forms to the central facility 
would be eligible to win a prize. 
E* Summary 

Section III has discussed the strategy and procedures 
for the implementation of the basic or core reporting and 
evaluation system in planning regions 1 and 9. The implemen- 
tation strategy developed and tested for these planning regions 
should work equally well throughout all of New York State 
including the big-six cities which are not included in the 

Preliminary discussions with respect to the concepts, justi- 
fications and organization of an occupational education follow- 
up sweepstakes have been held with occupational directors or 
guidance personnel at most BOCES in planning regions 1 and 9* 
Considerable enthusiasm has been expressed. However, before 
a sweepstakes could be implemented the ramifications of this 
approach will have to be examined by RRI and the State Educa- 
tion Department Division of Occupational Education Planning. 
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BOCES network. However, because problems of size could cause 
some unique difficulties for RESOE in the state's larger 
cities — particularly in New York City — RRI carried out a 
systems study and field test to determine the feasibility of 
implementing RESOE in New York City. This effort is described 
in the following section of this report. 
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V. Implementation of RESOE in New York City 
A. Overview 

During the past year, RRI carried out a number of re- 
lated tasks to determine the feasibility of implementing 
RESOE in New York City. RRI's tasks included: a) assess- 
ment of current procedures used in New York City for meeting 
mandated reporting requirements and for carrying out other 
processes which RESOE supports; b) determination of whether 
RESOE could provide the city with increased capability for 
executing reporting and evaluation functions in occupational 
education; c) modification of some of RESOE 's procedures so 
that the particular requirements of New York City can be met; 
and d) implementation of a limited RESOE field test. 

The RESOE feasibility study began with a series of 
meetings at the Board of Education. RESOE was explained to 
board staff members who are responsible for the administra- 
tion of occupational education and educational data processing. 
These meetings culminated in a presentation of RESOE concepts 
and capabilities to Chancellor Scribner on June 12, 1972. 
The Chancellor expressed considerable interest, and therefore 
authorized the continuation of work efforts with the proviso 
that RESOE be implemented in New York City in a manner which 
did not contradict current policies and practices concerning 
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the confidentiality of potentially sensitive student data, 
and redundant utilization of teacher time for meeting re- 
porting requirements. Through the coordinated efforts be- 
tween RRI and board staff members, the particulars of RESOE 
implementation were agreed upon and a field test was executed. 

This section of the report describes the three major 
components of RRI ' s work efforts in New York City: a review 
of current practices; a proposal made to the board for RESOE 
implementation; and the New York City RESOE field test. 
B. A Brief Review of New York City's Procedures for Meeting 
Mandated Reporting Requirements in Occupational Education 
Board of Education procedures for fulfilling mandated 
reporting requirements were emphasized in RRI ' s review, 
since the fulfillment of these requirements is the key func- 
tion carried out by RESOE during the initial stage of imple- 
mentation. 

!• Enrollment reports . The procedures for meeting the 
mandated reporting requirements in the academic high schools 
differ from the procedures employed in the vocational high 
schools. In the academic high schools, a student survey is 
carried out which is not totally unlike the RESOE enrollment 
survey described earlier in these pages. Each student com- 
pletes a single "Occupational Enrollment Record" card with 
the following critical information: 
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• School 

• Official olass 

• Name 

• Name of last attended elementary or junior high 
school 

• Indication of registration in courses for the 
physically handicapped 

• Indication of participation in cooperative educa- 
tion 

• Indication of planned occupational major in the 
high school in which the student is currently 
taking courses 

The students also help to determine their own stu- 
dent type (regular or disadvantaged) . A list of elementary 
and intermediate schools not eligible for assistance under 
Title I of the Elementary and Secondary Education Act is read 
by che teacher. If a student has attended one of these schools 
he is instructed to record that fact on the "Occupational En- 
rollment Record" card. Students from these schools are con- 
sidered to be regular, the other students are defined as 
being disadvantaged. 

The teacher collects the cards, circles a code let- 
ter to indicate student ethnicity, and then removes a perfo- 
rated tab on the card which contains the student's name. 
Removal of ^he tab serves to protect the anonymity of each 
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student's data* The cards are then sent to the board for 
data processing • 

At the vocational high schools, no student survey 
is conducted- Instead, the central administration is re- 
quested to report on forms provided by the State Education 
Department (OEP-1) , an unduplicated count of the number of 
students in each occupational education program. The number 
of students in each program is to be partitioned by grade, 
race and student type (disadvantaged, handicapped, regular) . 
The definition of disadvantagement is based upon socio- 
economic criteria such as eligibility for free lunch program, 
student's address in an obviously poor neighborhood, etc. 
The completed forms are sent to a central location where they 
are aggregated over schools, so that a single form per pro- 
gram is produced. 

2. Follov;-up reports . New York City's follow-up sur- 
vey is handled similarly for graduates from the academic and 
vocational high schools- In the spring, students who have 
taken occupational courses fill in their mailing address on a 
double post card with the following information: 

• School from which students expect to graduate 

• Major occupr.tional or vocational course 

• Indication of whether students ever had a coopera- 
tive or work-study job 
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• Sex 

o Name of school attended before high school 
The rest of the questions concern employment and 

educational status. These are answered by the students when 

they receive the post card in the fall: 

• Name and home address 

• Employed full time in a field for which trained 
« Employed full time in a related occupation 

• Employed full time in unrelated occupation 

• Kind of work done 

• Wages 

• Seeking employment 

• Schools attended since graduation 

• Military status 

• Reasons for not working (if unemployed) 

The student then mails the post card back to his 
school. The data on the cards are ultimately aggregated to 
complete the follow-up report required by the State Education 
Department. 

3. Evaluation of New York City's reporting procedures . 
The Board of Education has implemented reporting procedures 
that are organized exclusively for the purpose of meeting 
two state reporting requirements: a report of occupational 
enrollments; and a report of occupational completions. When 
considered in this context, there is little in the board's 
procedures which could be characterized as flawed. Picayune 
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problems notwithstanding, the board is able to submit reports 
on schedule to the State Education Department that are no 
less valid than the reports submitted by other education 
agencies throughout the state. 

In the category of problems which are resolvable 
without substantial alteration of current practice, the most 
serious is the manner in which the board defines student dis- 
advantagement . Under the Vocational Education Amendments of 
1968, disadvantaged students are defined as those "... who 
have academic, socio-economic, or other handicaps that prevent 
them from succeeding in the regular vocational education pro- 
gram. " 

The State Education Department's Division of Occupa- 
tional Education Planning has helped to operationalize the 
federal intent by providing the following definition for re- 
porting purposes: 

Disadvantaged students are those who 
have academic, socio-economic, cultural, or 
other handicaps that affect their ability 
to succeed in an occupational education 
program designed for persons without such 
handicaps, and who for that reason need 
special educational assistance or a modi-- 
fied cacupational program. 

Unlike all other federal legislation concerning the 
disadvantaged, ^ 68 emphasizes the consequence of a given 
socio-economic condition rather than the condition itself. For 

* SEC. 122(a)(4)(A), VEA68. 
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example, students A and B may both come from a lower socio- 
economic stratum. Student A requires special assistance or 
a modified program. Student B does as well or better than 
his middle class colleagues without receiving special assist- 
ance. Under VEA68 only Student A should be identified as 
disadvantaged, since only student A requires special assist- 
ance. Under the board's current procedures both students A 
and B would be defined as disadvantaged because the board's 
definitions emphasize socio-economic factors rather than the 
actual consequences of these factors upon s-^udent performance. 
Under the board's current procedures, a student is disadvan- 
taged if he is eligible for free lunches, his home is in an 
obviously poor neighborhood, or if he comes from an interme- 
diate or elementary school which is eligible for funds under 
ESEA Title I. 

To the extent that there are students in New York 
City like student B, the board is reporting too many students 
as being disadvantaged. This problem may be resolved by 
having teachers or guidance personnel identify students who 
require and are receiving special assistance or a modified 

It should be stated that the problem of identifying disad- 
vantaged students under VEA68 is not more acute in New York 
City than in other locations. Neither VEA68 nor its subse- 
quently published rules and regulations are particularly 
helpful in defining what is meant by a disadvantaged student. 
The legal and functional ramifications of identifying the 
disadvantaged under the Vocational Education Amendments of 
1968 were investigated by RRI. An oral report was made to 
the State Education Department on December 18, 19 72. 
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program in occupational education because of socio-economic, 
academic, cultural or other (nonphysical) handicaps. These 
students and no others should be reported as disadvantaged. 

Other minor problems with the board's procedures 
include some duplication of work efforts. For example, since 
no file (computerized or otherwise) exists m which students 
are designated as disadvantaged on the enrollment report, the 
data must be reassembled for the follow-up report. Finally, 
student frequencies in the follow-up report cannot be readily 
partitioned by ethnicity (as required by the State Education 
Department) because there is no ethnic designation on the 
follow-up post cards, and because the ethnic designations pro- 
vided by teachers on students' enrollment cards cannoi- be 
associated with the student's follow-up returns under the 
board's procedures. 

The major problem with the board's procedures is 
that a substantial amount of professional labor is consumed 
relative to what is ultimately produced. The city's proce- 
dures cannot easily be augmented to add other reporting or 
evaluation functions. This problem exists because of the 
essentially ad hoc nature of the procedures i:\ use. The pro- 
cedures serve only to meet two specific occupational education 
reporting requirement-s . The implementation of RESOE and some 
minor changes in the present procedures for fulfilling the 
two occupational education reporting requirements would reduce 
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the time required to prepare the mandated reports, as well as 
provide the board with the basic capability to systematically 
assess program effectiveness in occupational education, to 
produce cost studies and longitudinal follow-ups , to provide 
repOiTts for local constituencies (snch as parent groups, 
labor unions, employers, community school boards) and to ef- 
fectively coordinate a variety of existing administrative 
processes. The recommended changes and RESOE implementation 
steps are described in the next section. 
C . Recommended Procedures for RESOE Implementation 

The procedures v;hich are described below result in the 
tx.lfillment of occupational education reporting requirements 
and the establishment of a basic SEED-maintained student file, 
the contents of which are similar to the files established for 
planning regions 1 and 9. The procedures have been worked out 
by RRI with the support and suggestions of administrators at 
the Board of Education. The recommended system is illustrated 
in Figure 6 . 

1. Enrollment reports . As in planning regions 1 and 9, 
RESOE implementation begins with the dissemination of materials 
(instructions and forms) to schools within the city which of- 
fer occupational education. There are two essential forms; 
a course description form which is completed for each occupa- 
tional course by a school secretary or clerk, and a student 
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Fig. 6. Flow diagram of procedures 
for the implementation of the Re- 
porting and Evaluation System in 
New York City. 
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enrollment card which is completed by each student foi each 
course he takes. 

The secretary indicates the name of the school, 
local school code, the name of the occupational course, and 
the room where the course is taught. The secretary also 
fills in the standard occupational program code for the 
course, she also indicates whether the class is regular or 
part of a cooperative work experience program. The nane and 
local address of the person who has been designated to re- 
ceive the envelope after the students have completed their 
enrollment cards are also entered on the form. 

On a perforated tab attached to the enrollment 
card, the student writes the name of his school, the date, 
and his own name. The student then enters his initials, date 
of birth, sex, grade, cooperative education status. Social 
Security number, and indicates whether he has completed the 
form in another course. 

The Social Security number is requested in order to 

obtain a unique numeric identifier for e?ch student that can 

be used for computer merging operations and for related sys- 

* 

tems applications. In the near future, it will not be 
difficult to obtain Social Security numbers from most 

* 

The Targeting System for Occupational Education requires 
Social Security numbers of students completing occupational 
education courses . 
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students. In the meanwhile, a unique numeric identifier can 
be created from the location code for the student's school, 
his initials, date of birth, and sex. 

After the students complete the forms, the teacher 
collects the forms and indicates student ethnicity and stu- 
dent type. Ethnicity is indicated by circling a letter (I, 
N, 0, S or A) . The teacher also circles a letter (R, D or 
to indicate student type (regular, disadvantaged, handicapped) 
The following definitions were developed with the board to be 
used by teachers for defining the disadvantaged and for de- 
fining the handicapped. A disadvantaged student is one who 
requires special assistance or a modified course of study in 
the siobject for which the enrollment card is bein j completed, 
because of academic, 5ocio"-econo:aic , or cultural handicaps. 
A handicapped student is one '-no requires special assistance 
or a modified course of study in the subject for which the 
enrollment ccird is being completed, because of mental 



Federal legislation has recently beexi enacted which re- 
quires students to have Social Securit^^ numbers upon entering 
school. Moreover, the Social Security Administration has co- 
operated with school administrators in obtaining Social 
Security nunbers for occupational students. After a presenta- 
tion in school is made by a representative of the Social 
Security Administration on "the Social Security System in the 
world of work," applications are given to students for imme- 
diate completion. The applications are then sent eri masse to 
the local Social Security office. The office prepares cards 
and returns them to the scnool for distribution to students. 



F/242-5-00 - 72 - 

o 83 
ERLC 



retardation, hearing difficulty, deafness, speech irapairment, 
visual handicap, serious emotional disturbance, crippling, or 
other health impairment. 

After the teacher indicates student ethnicity and 
student type, the perforat.9d tab containing the student's 
name is detached, the enrollment cards are replaced in the 
envelope, and the envelope is returned to the individual des- 
ignated to receive it. After each occupational course is 
surveyed, and all envelopes are collected at a particular 
school, they are sent to the Board of Education. 

The forms are then sent to the RESOE central fa- 
cility where the enrollme.it data are punched and SEED-maintained 
student files are created. SEED is then used to generate the 
enrollment report required by the Scate Education Department. 
I'ldividual schools are also sent an enrollment report. 

2. The follow-up report . A modified version of the 
post card system currently used by the board may be used 
within RESOE. In late April, each graduating student enters 
his projected fall address in the correct place on a double 
post card. The student also provides sex. Social Security 
number, school code and perhaps some other identifying infoima- 
tion on the half of the double post card which does not contain 
his name and address. These cards are also completed by oc- 
cupational dropouts throughout the year and by certain stu- 
dents in adult courses. In June, the cards are sent to the 
Board of Education; and then to the central facility. 
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SEED is then used to create a file of students to 
be followed up. This is accomplished by entering the identi- 
fiers of students to be followed up and adding to it the 
information on these students available in the previously 
created student enrollment file. After the follow-up file is 
created, all sensitive data (e.g., student ethnicity and stu- 
dent type) are purged from the permanent enrollment file, 
yfhis step is carried ^at to fulfill a New York City policy 
of not retaining sensitive data on individual students. 

Some small proportion of students who fill out post 
cards in the spring would not be foxind in the SEED-maintained 
enrollment files. This does not adversely affect RESOE's 
ability to carry out the follow-up study. If requested to do 
so, the central facility produces listings of student comple- 
tions with miscing enrollment data, so that local school ad- 
ministrators interested and able to provide missing enroll- 
ment information can do so. 

In the fall, the central facility sends the post 
cards out. As the follow-up cards are returned, they are 
merged into the data base. If a second mailing is to be car- 
ried out for students who do not respond to the first follow- 
up questioFxnaire, the central facility uses its SEED-maintamed 
file to determine which students have not responded and to 
generate labels for a second mailing. 
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At the last possible date permitted by SED, the 
state mandated follow-up report is generated by SEED through 
the use of the student follow-up file. All students who have 
not returned their cards would be listed as "status unknown" 
on the state report. Reports are also generated for each 
participating school. As in follow-up studies throughout the 
rest of the state, the central facility would calculate how 
representative the students who returned follow-up data are 
as a sample of the entire population of students who are being 
followed up. All post cards that are returned after the 
closing date for filing the state report are keypunched, and 
the data entered into the data base for any subsequent analy- 
ses requested by the board or by individual schools. As in 
the case of the enrollment files, the follow-up files are 
pu.-ged of sensitive data (e.g., student ethnicity and student 
type) to meet a board policy of not retaining such information. 
D. Field Test of Enrollment Procedures 

After the procedures described in the previous section 
were developed by RRI , and accepted by Board of Education 
staff members, a field test of the RESOE procedures was car- 
ried out. The field tGst was concerned with the enrollment 
portion of the system. The RESOE enrollment processes were 

* 

For comparative purposes the field test enrollment procedures 
includ'id two definitions of disadvantagement. Under the first 
definition, a student was designated as disadvantaged if he 
came from a school eligible for ESEA Title I assistance. Under 
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field tested in one vocational and two academic high schools. 
The forms and instructions for this field test are contained 
in Appendix 3 of this report. The data from the field test 
are still being analyzed. 

Although final conclusions with respect to the success 
of the enrollment procedures in New York City must await the 
completion of data analysis, a preliminary review cf the data 
indicates that the enrollment procedures were successful, and 
that it would therefore be possible to provide New York City 
with the flexible reporting and evaluation apparatus that 
RESOE offers. 



the second definition, the student was disadvantaged if he re- 
quired special assistance or modified instruction because of 
academic, socio-^economic or cultural handicaps. The simultaneous 
utilization of the two deflations allow for an empirical assess-- 
ment of the differences and how provided information concerning 
enrollment and follow-up reports will ultimately be affected 
by changing definitions. However, when RESOE is implemented in 
New York City, the second definition will most likely be utilized, 

A supplementary report on the RESOE New York City field 
test will be given to the State Education Department after all 
data are analyzed and final conclusions are made. 
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VI. Utilization of RESOE by Regional and 
Local Education Agencies 
Most of this report is concerned with the initial stage 
of RESOE implementation, which involves the establishment of the 
basic SEED-maintained student files, and the utilization of 
these files to fulfill state reporting requirements. However, 
the full value of RESOE is realized when the system is used 
to meet state reporting requirements while simultaneously 
supporting additional reporting, evaluation, and administra- 
tive processes for occupational administrators and guidance 
personnel. 

Although RESOE distributes and reduces the work asso- 
ciated with meeting reporting requirements, the time expended 
by local personnel to provide the central facility with the 
data it needs to create files and generate mandated reports 
is hardly trivial. By using RESOE for ^additional purposes, 
the full power of flexible computer-maintained files is 
realized. The time expended to create those files can be 
apportioned over a variety of functions, and total personnel 
time is genuinely conserved. 

The utilization of RESOE for additional reporting and 
evaluation purposes will be initiated at the discretion of 
local and regional occupational educators. Thus sume educators 
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might wish to use RESOE for analysis of skills mastery, others 
might wish to use RESOE to produce the data for a regional 
"state of occupational education" report, others may apply 
RESOE for the analysis of sophisticated follow-up research, 
and others might be interested in program costing, or a 
variety of other administrative- processes. On the other hand, 
some occupational educators may \'ish to use RESOE only to meet 
mandated reporting requirements, and to apply alternative ap- 
proaches for fulfilling other reporting and evaluation func- 
tions . 

Occupational educritors will be able to make an informed 
choice over such matte^-'s only after they have become familiar 
with the basic concepts underlying New York State's RESOE, 
and after RESOE has become able to fulfill their requirements 
and specifications. During the past year, RRI carried out a 
number of tasks to make RESOE more comprehensible to occupa- 
tional administrators and more responsive to their requirements. 
These v/ork efforts are discussed in this section. 
A. A Layman's Guide to the Reporting and Evaluation System 

During the past year, RRI has developed a guide to RESOE. 
The purpose of this guide is to acquaint occupational educa- 
tors with the concepts and capabilities of RESOE, and to 
provide a set of procedures which allow occupational educators 
to have access to their SEED-maintained data at the central 



F/242-5-0 0 



- 78 - 



facility. The layman's guide appears as Appendix 1 of this 
report. 

B. Bulletins Concerning System Additions and Changes 

Procedures need to be found to supplement the layman's 
guide so that occupational educators will be aware of system 
additions, improvements, and also of any problems of general 
significance which have emerged in the course of using FffiSOE 
for a variety of purposes. Some relatively formal means of 
communication will be required when RESOE has been implemented 
in several occupational planning regions. RRI has investi- 
gated alternative means of disseminating information to RESOE 
participants. Two forms of information dissemination seem to 
be most appropriate. Information of a relatively technical 
nature can be sent as supplements to the layman's guide. 
After a number of supplements have been sent, or when any 
major revisions in central facility procedures or in SEED are 
implemented, a revised version of the guide should be produced. 
Nontechnical information of general interest can be sent by 
the central facility to the regional coordinators in the form 
of press releases. These releases may be included in existing 



After the contents of the layman's guide are approved by the 
State Education Department's Division of Occupational Education 
Planning, the guide can be produced as a brochure for distribu- 
tion to occupational educators at sites where RESOE is being 
implemented. 
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BOCES and local education agency publications if the regional 
coordinator finds it appropriate to do so. 

C. Revisions in RESOE^s Stage I Implementation Procedures ; 
Changes in Coding Procedures 

As indicated elsewhere in these pages, the primary objec- 
tive of the RESOE Stage I implementation process is the es- 
tablishment of basic student files which may be used to ful- 
fill mandated reporting requirements. A number of minor 
changes can be effected which will result in a student data 
base which fulfills the mandated reporting requirements, and 
is also more capable of serving the needs of local administra- 
tors . 

An example of a change which should be implemented is in 
the course coding procedures. In the current procedures, 
occupational courses are coded with a subset of program codes 
developed originally by the U. S. Office of EduCs-tion. The 
mandated enrollment and follow-up rc ports must make use of 
these codes. However, occupational administrators frequently 
do not use these codes in their own administrative practices. 
They have often developed a local coding system which they 
have found to be more useful for their particular set of 
course offerings. If the RESOE data base is keyed only by the 

A listing of these codes may be found in Appendix 2 in a 
booklet entitled Program Classifications for Occupationa l 
Education . 
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federal code system, local administrators have difficulty in 

compiling their course or student data with respect to course 

categories that are meaniigful to them. Very frequently, a 

single course may be categorized under several federal codes. 

In other instances, courses which are quite different in the 

view of local administrators must both be labeled with the 

* 

same federal code. 

Therefore, as part of the Stage I implementation proce- 
dures, occupational educators should have the opportunity to 
enter their own local codes in addition to the federal codes 
or to provide the central facility with a table which "cross- 
walks" local course and federal program codes. If these 
codes are then entered into the basic student enrollment and 
follow-up files, enrollment and follow-up reports can be 
generated which will have more local significance. 
D. A Regional State of Occupational Education Report 

During the past year, RRI has initiated discussions with 
occupational educators concerning their interest in an annual 
"State of Occupational Education" report which would contain 
summaries of occupational activities, student performance, 
etc. that were generated by RESOE. Some occupational 

* 

An example of this situation occurs in New York City where 
one federal code must be used to describe two distinct types 
of stenography courses. One type is for students with ade- 
quate English skills, and the other type is for students 
without adequate English skills. 
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^ administrators have expressed interest in this sort of report, 

while others would prefer to use RESOE for the generation of 
Riore frequent (e.g., quarterly) reports. Based upon dis- 
cussions with occupational personnel, the following informa- 
tion might be included in an annual state of occupational 
education report or in more frequently generated shorter 
reports: 

® Results of follow-up studies; 

• student enrollments; 

• reports of student performance; 

® reports of utilization of BOCES occupational education 
services by member school districts; 

• reports of placement activity; 

• reports of guidance activity; 

execution and reports of surveys of students * occupa- 
tional interests; 

• reports of the manpower requirements of local and 
regional employers; 

o reports of faculty coverage and experience; 

• reports of the utilization of services for special 
student groups such as the handicapped and the disad- 
vantaged. 

In addition to this quantitative information, the "State 
of Occupational Education Report" also needs to contain de- 
scriptions of new course offerings, equipment acquisitions, 
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announcemeKts of any special facilities which have been re- 
cently introduced, and reports of special community service 
projects which are being carried out* 
Summary Statement 

In RRI ' s view, the interest in the system shown by local 
and regional education agencies is one of the most important 
results of the coordinated efforts carried out during the 
past year by RRI and the State Education Department Office of 
Occupational Education. This interest will support the sus- 
tained development of the system so that it becomes continuously 
more useful to all users. Therefore, the implementation of 
RESOE in additional locations must be coupled with appropriate 
work efforts to ensure that the system serves local require- 
ments. The reporting and analysis request system de- 
scribed in the layman's guide must be tested and revised if 
necessary. Inexpensive remote data entry procedures for ac- 
cessing centrally stored information should also be developec' 
and field tested. Finally, a substantive plan must be devel- 
oped for integrating within RESOE an increased number of 
currently independent local operating functions so that 
RESOE continues to conserve time and money. 
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